Skip to main content

Query Analytics Installation

Setting up Query Analytics for Elasticsearch and Opensearch​

General information and architecture​

In order to see Query Analytics in Pulse, you will need to set up a special proxy between your querying application (be it application code, logstash, or anything else) and ES/OS.

This proxy will perform the search requests as requested and return the responses, while additionally logging metadata from the request and its body, and sending those to Pulse.

The proxy does not differentiate between different types of requests - meaning you should ONLY send search requests to it, while still sending other requests directly to the cluster.

To provide permission for the proxy to access your cluster, you will need to go through the respective ES/OS configuration for the user pulse-monitoring-collector - see here for Elasticsearch and here for Opensearch.

You will also need a unique access token received from BigData Boutique, which is the same one used for the Pulse Agent.

Keep in mind that obfuscation of the search query is done after it reaches the pulse ingestion endpoint.

Setting Up a proxy to send Query Analytics data​

  1. As a prerequisite to setting up Query Analytics, you should contact Pulse support to receive the ingestion endpoint to use in your config file.

  2. Start by creating the following configuration file, call it config.yml and set the following values (leave the others as is):

under audit-log:

  • auth-token - a unique access token received from BigData Boutique

under cluster:

  • type - The type of cluster you are connecting to. Currently, only elasticsearch and opensearch are supported.
  • host - An Elasticsearch/Opensearch endpoint for the initial connection of the proxy.
  • username and password - these are the credentials for the pulse user. They are only used during the initial connection to Elasticsearch/Opensearch. For the actual requests the credentials passed in the request are used.
  • reject-unauthorized - If you are using a self-signed certificate, you can set this to false to ignore certificate validation.

under proxy:

  • port - The port on which the proxy will listen for requests.
  • https.key and https.cert - If you want to use HTTPS, you can provide a key and cert file. If you don't provide these, the proxy will use HTTP.
audit-log:
  auth-token: "your-pulse-auth-token-here"
cluster:
  type: elasticsearch
  host: "http://host.docker.internal:9200"
  username: pulse-monitoring-collector
  password: L0ng-r4nd0m-p@ssw0rd
  reject-unauthorized: true
proxy:
  port: 9201
bulk-max-size: 500
flush-interval: 10000
  1. Make sure the proxy machine has access to the Elasticsearch/Opensearch cluster and that the querying application has access to the proxy machine, to the port you choose to use for the proxy. The proxy machine should also be able to access https://pulse.support.
  2. If the machine cannot download docker images from r.bigdataboutique.com, you'll need to download the image to a separate machine and then import it to the proxy machine.
  3. run the following command from the folder that has config.yml on the proxy machine: docker run -p 9201:9201 -v `pwd`/config.yml:/etc/pulse-search-gateway/config.yml --name query-analytics-proxy r.bigdataboutique.com/search-gateway:latest
  4. Test that you can send requests to the proxy address and get responses.
  5. Verify that your queries are correctly appearing in Query Analytics.